[00:08.580 --> 00:14.900]  Good afternoon. It's a wonderful honor to be here today. Tonight, or this afternoon,
[00:14.900 --> 00:20.100]  I'm going to be talking about Capture the Flag. Capture the Flag is a contest we like
[00:20.100 --> 00:27.860]  to run at DEF CON. DEF CON, you're here. We've run Capture the Flag at DEF CON in Las Vegas
[00:27.860 --> 00:35.400]  for several years now. In 1996, the first Capture the Flag game at DEF CON was held.
[00:35.880 --> 00:41.420]  And in the year 2000, so after four years, they formalized how it was run. So for the
[00:41.420 --> 00:46.580]  first few years, how Capture the Flag worked at DEF CON was players would either bring
[00:46.580 --> 00:51.640]  software that they wanted somebody to hack for them, or they would hack somebody else's
[00:51.640 --> 00:57.840]  software. And the human judges that were authenticating and running the contest had to make the
[00:57.840 --> 01:02.220]  these decisions every time something got hacked, whether it was easy, whether it was
[01:02.220 --> 01:08.380]  hard, who got the points, and it was very, very confusing. So starting in the year 2000,
[01:08.380 --> 01:13.680]  the DEF CON goons running Capture the Flag created rules, and they created services.
[01:14.640 --> 01:20.800]  So starting in 2002 and running through 2004, a group called Ghetto Hackers took over Capture
[01:20.800 --> 01:25.180]  the Flag. And this is kind of how Capture the Flag became a name brand event, where
[01:25.180 --> 01:29.480]  the Ghetto Hackers had a different brand name, slightly different from DEF CON. They
[01:29.480 --> 01:36.920]  had continuity from year to year. Starting in 2005, a new team called Kenshoto took over.
[01:36.920 --> 01:41.300]  And this is also when I started playing Capture the Flag. Kenshoto had a much more difficult
[01:41.300 --> 01:55.160]  game that made a lot more people more interested in Capture the Flag. Starting in 2009, the
[01:55.160 --> 02:01.020]  next team called DD Tech and started running Capture the Flag themselves. And then, starting
[02:01.020 --> 02:06.760]  in 2013 and running up until last year, my team, the legitimate business syndicate, took
[02:06.760 --> 02:12.980]  over. And starting literally tomorrow, the new team, Order of the Overflow, is taking
[02:12.980 --> 02:18.920]  over running DEF CON Capture the Flag. So the way we think about Capture the Flag and
[02:18.920 --> 02:23.420]  how Capture the Flag works with DEF CON is that there are two distinct formats. This
[02:23.420 --> 02:28.680]  format is Jeopardy format, named after the U.S. game show. And how this works is teams
[02:28.680 --> 02:33.440]  pick a question or a challenge from this grid. They receive a prompt and solve the challenge
[02:33.440 --> 02:38.820]  for some points. The main part of this game is the scoreboard that we just saw. On the
[02:38.820 --> 02:43.900]  scoreboard, you find a prompt. In this case, the prompt is giving us a network address,
[02:43.900 --> 02:50.980]  including a port number and a file to download. From there, we solve the challenge. In a lot
[02:50.980 --> 02:55.300]  of cases, these challenges involve a lot of assembly programming or reading assembly.
[02:55.300 --> 03:02.020]  And they involve Python. Once the challenge has been solved, you return the answer to
[03:02.020 --> 03:07.560]  the question to the scoreboard for points. And then you go on to the next challenge.
[03:07.620 --> 03:11.840]  Right here, we see a handful of challenges that have been solved by somebody with different
[03:11.840 --> 03:16.240]  point amounts. And one challenge that has not been solved yet and that will give the
[03:16.240 --> 03:23.180]  person who solves it scoreboard control. So Defcon Capture the Flag qualifiers is one
[03:23.180 --> 03:27.600]  of the most famous Jeopardy-style CTFs in the world. But I'm going to talk about a challenge
[03:27.600 --> 03:36.100]  from a different event, the Shaw 2017 Capture the Flag contest. So Shaw 2017 was an outdoor
[03:36.100 --> 03:41.820]  computer festival held in the Netherlands last August, the week after Defcon. And I
[03:41.820 --> 03:45.940]  went there and had a wonderful time. And part of the wonderful time was a contest
[03:45.940 --> 03:51.340]  that the Dutch team Eindhoven ran at this event. I'm going to talk about a challenge
[03:51.340 --> 03:57.380]  called Asby. With Asby, we get a file. This is the prompt that has a little bit of backstory
[03:57.380 --> 04:05.500]  where their team member named Asby got a reputation for solving challenges the wrong way. So once
[04:05.500 --> 04:11.800]  I downloaded this file, I discovered it was a Windows executable. And because I just,
[04:11.800 --> 04:15.820]  like, downloaded this executable randomly from the internet, I decided to run it and
[04:15.820 --> 04:22.780]  figure out what I had to do. So the goal with this is we need to enter in the flag, and
[04:22.780 --> 04:28.160]  the executable will tell us if it's correct or not, and starting from here, how many characters
[04:28.160 --> 04:34.500]  are correct. So I need to guess the correct input. So I have three ways that I can do
[04:34.500 --> 04:39.920]  this. I can either reverse engineer a Windows binary, and I'm not very good at reverse engineering
[04:39.920 --> 04:44.740]  and I don't usually use Windows, so that was kind of out. I could guess each character
[04:44.740 --> 04:50.440]  by hand, which looked like it would take a long time, or I could write a program, which
[04:50.440 --> 04:58.080]  I foolishly believed would go really, really quickly. So I chose to write the program.
[04:58.580 --> 05:04.740]  And this is the program. It's a little Ruby script, and mostly what it does is it opens
[05:04.740 --> 05:11.180]  the wine emulator with the Asby executable. I hand guessed a bunch of the flag before
[05:11.180 --> 05:15.760]  I tried writing the program, so I have that part of the flag at the top. And then we just
[05:15.760 --> 05:20.660]  loop over and over, guessing which of our flag characters goes into the executable and
[05:20.660 --> 05:26.300]  tells us it's correct. If we get to a point where something weird happens, we print out
[05:26.300 --> 05:32.980]  what we have so far and exit. In theory, after a long time writing this program and switching
[05:32.980 --> 05:36.720]  back and forth between, oh, I can't program computers anymore, I'm just going to do this
[05:36.720 --> 05:42.440]  by hand, I eventually got the solution. The solution is right here, right above the stack
[05:42.440 --> 05:48.720]  trace, which is cool. I solved it, I got some points, the organizers gave me some beverages
[05:48.720 --> 05:53.720]  while I was sitting there, and I had a lot of fun. So that's Jeopardy style. You get
[05:53.860 --> 05:59.020]  a challenge, you solve it, and you get some points. This is the visualization from our
[05:59.020 --> 06:04.680]  finals game, which is an attack-defense format capture the flag. It's very, very different.
[06:04.680 --> 06:11.740]  Here we see teams shooting successful attacks at each other in a soundboard-themed, or like
[06:11.740 --> 06:18.020]  audio mixer-themed event. So how this works is all these steps happen at the same time.
[06:18.020 --> 06:22.840]  Teams are reverse engineering the challenges or the services. They're patching the flaws
[06:22.840 --> 06:28.820]  in the services at the same time they're trying to exploit other teams' services. And in all
[06:28.820 --> 06:34.800]  of this, they're trying to not break their own services to lose points. So here's a small
[06:34.800 --> 06:39.660]  example of how attack-defense works. At the top, we have the score bot, or the scoring
[06:39.660 --> 06:45.120]  system of the capture the flag game. On the left, we have PPP's service called AppMail,
[06:45.120 --> 06:52.860]  and on the right, we have a team called Shellfish. So, every round, the score bot deposits a
[06:52.860 --> 06:58.480]  new flag or a new piece of random data in PPP's AppMail service, and everybody else's
[06:58.480 --> 07:07.260]  services as well. Sometime during the round, Shellfish manages to send some network traffic
[07:07.260 --> 07:12.640]  to AppMail that causes it to disclose this flag, or as we call it, Shellfish has stolen
[07:12.640 --> 07:19.320]  the flag. Then, they take that flag, submit it to the score bot, or they redeem it to
[07:19.320 --> 07:26.560]  the score bot, and get some points. In the meantime, score bot is checking PPP's AppMail
[07:26.560 --> 07:31.680]  service to make sure it's working okay. Can it send messages? Can it receive messages?
[07:31.820 --> 07:38.960]  Is it happy? And, if PPP has broken their AppMail, if they've, you know, turned it off
[07:38.960 --> 07:44.840]  or deleted it, or just written a bad patch that makes it do things wrong, that's a failed
[07:44.840 --> 07:49.200]  availability check. And we assume that if there's a failed availability check, that
[07:49.200 --> 07:55.240]  Shellfish is also not able to steal from it, so we make a failed availability check lose
[07:55.320 --> 07:59.600]  a whole lot of points. And, again, this is the format we used for Defcon Capture the
[07:59.600 --> 08:05.840]  Flag finals. Here's an example. It's a service called Rubix. This is a service we put in
[08:05.840 --> 08:13.880]  place last year. How it works is teams connect to the Rubix service on another team, and
[08:13.880 --> 08:17.920]  it simulates this Rubix puzzle cube where you rotate the cube and try and get all the
[08:17.920 --> 08:23.540]  colours on the same side. But it's a virtual version of that. Teams submit instructions
[08:23.540 --> 08:30.520]  to rotate the cube into a valid configuration, and once the Rubix cube is happy, it executes
[08:30.520 --> 08:39.200]  the instructions as if they were CPU instructions. Which is kind of clever. So we're going to
[08:39.200 --> 08:43.720]  follow along with a write-up from a team called Lab Rats. They publish this write-up and we
[08:43.720 --> 08:50.000]  can follow along. So the first steps to solve this, the computers we used for Defcon CTF
[08:50.000 --> 08:54.780]  finals last year was one we made up, and it used 9-bit bytes instead of 8-bit bytes
[08:54.780 --> 08:59.820]  to make it miserable for teams. So they first had to write software that could deal with
[08:59.820 --> 09:07.160]  9-bit traffic, which nobody has normally. Then they had to figure out what parts of
[09:07.160 --> 09:11.680]  these binaries were part of the standard C library, and what parts of these binaries
[09:11.680 --> 09:16.660]  were which functions in the C library. After that, they can figure out how the main gets
[09:16.660 --> 09:21.500]  called and what the part of the program that they care about actually is. Once they've
[09:21.500 --> 09:26.440]  done all that work, which took several hours, they could actually start analyzing what the
[09:26.440 --> 09:31.620]  service does. So what are they analyzing? How is it supposed to work? What is the benign
[09:31.620 --> 09:37.340]  traffic supposed to do? Once they figure that out, how can they attack it? How can they
[09:37.340 --> 09:42.080]  make the service release its secrets? And after they figure that out, they can try and
[09:42.080 --> 09:47.060]  figure out how to defend it. How can we prevent the service from revealing its secrets to
[09:47.060 --> 09:53.420]  other teams? So all this is in support of the game goals. You get points by capturing
[09:53.420 --> 09:59.180]  flags, you lose points by having flags captured, and you lose lots of points by failing availability
[09:59.180 --> 10:06.020]  checks. It's extremely complicated, it's extremely frustrating, but it's a lot of fun. For both
[10:06.020 --> 10:13.320]  organizers and most importantly, it's fun for players. But I should go back. Every kind
[10:13.320 --> 10:17.780]  of Capture the Flag game is extremely ambitious. Jeopardy style, they're extremely ambitious
[10:17.780 --> 10:25.000]  to run, they're difficult to play on purpose. Attack defense, also difficult. So some of
[10:25.000 --> 10:29.640]  our goals for Capture the Flag games are making them run smoothly, making them a fair contest,
[10:29.640 --> 10:34.280]  and having fun challenges. If the game doesn't run smoothly, players are going to get frustrated
[10:34.280 --> 10:39.500]  and find something else to do. If it's not fair, players aren't going to put in a legit effort.
[10:39.620 --> 10:44.040]  You know, if the same team is always going to win because they're friends with the organizers,
[10:44.400 --> 10:50.800]  why play? And if the challenges aren't fun, then the challenges aren't fun. So let's talk
[10:50.800 --> 10:55.420]  about running the game smoothly. And just like any big ambitious project, running a Capture the
[10:55.420 --> 11:01.600]  Flag game project starts early. And the most important part of it is who you're running it
[11:01.600 --> 11:08.220]  with, who is on the organizing team. So legitimate business syndicate is about half of us are friends
[11:08.220 --> 11:13.240]  from university that competed together then, and about half were people that worked together in
[11:13.240 --> 11:20.640]  2012. So here's my friend Gino, kind of the team captain and I. Twelve years ago, we were
[11:20.640 --> 11:25.600]  participating in a college level contest, and we just solved a difficult problem. And we were very,
[11:25.600 --> 11:30.920]  very happy about it. And also a little bit weird from being in a windowless room all day.
[11:32.820 --> 11:39.740]  So how did we start to run DEF CON CTF? We found out during closing ceremonies at DEF CON 2012,
[11:40.160 --> 11:46.160]  that the previous organizers, DD Tech, were stepping down. In December, Gino realized that,
[11:46.160 --> 11:53.100]  hey, I should make a team that runs CTF because that's my dream. So he started talking to us,
[11:53.100 --> 11:59.080]  and I remember at a chicken wing restaurant, the day after Christmas is when he told me about this,
[11:59.080 --> 12:05.740]  and I was in. So in February 2013, we all got together in the same room, wrote down a proposal
[12:05.740 --> 12:11.340]  for how we wanted the game to work, and found out in March that the proposal got accepted. And then
[12:11.340 --> 12:18.580]  the real work began. So what is our group made out of? So about three quarters of the group are
[12:18.580 --> 12:23.600]  people that I would normally call reverse engineers. But people, you know, people aren't
[12:23.600 --> 12:28.300]  just their skills. And all these people have different specialties. So we have a radio
[12:28.300 --> 12:34.500]  specialist who did a lot of work on our 2014 Challenge Badger. We have several people who
[12:34.500 --> 12:39.900]  were good just with computer hardware in general, which was really useful for 2015, the year we
[12:39.900 --> 12:46.360]  brought, I think, 40 different computers to Las Vegas. And in 2017, the game ran on the work of
[12:46.360 --> 12:55.020]  our esoteric computing expert for the Clemency system. But besides reverse engineers, my friend
[12:55.020 --> 13:02.340]  Salir is just a genius when it comes to computer infrastructure of any kind. Networking. He can
[13:02.340 --> 13:06.540]  type commands into routers faster than I could just mash my fingers on a keyboard. I don't
[13:06.540 --> 13:12.780]  understand it. I worked on the database-backed web application because that's something I really
[13:12.780 --> 13:19.100]  enjoy. But the most important part is that people grow and change. I'm not the same person I was
[13:19.100 --> 13:24.280]  five years ago. None of us are. So whenever somebody, you know, grows into a new skill set or
[13:24.280 --> 13:30.100]  gets excited about something, don't say you're the web app person. Just deal with that. Like, if
[13:30.100 --> 13:36.380]  they're excited about something, that's a big benefit for everybody. And that means that players'
[13:36.380 --> 13:41.880]  role or team members' roles also grow and change over time. So if you are putting together a
[13:41.880 --> 13:47.140]  capture-the-flag team, either for playing or for organizing, think about who you know and who you
[13:47.140 --> 13:53.340]  know that you trust and who do you like? Who do you want to hang out with a lot? So once you've
[13:53.340 --> 13:56.860]  figured out who's on the team, you need to be able to talk to your team. And that's what
[13:56.860 --> 14:03.820]  communication is for. We, after the first year, we ended up using Slack or the chat program Slack
[14:03.820 --> 14:09.300]  for asynchronous chat basically all the time. It's great because you don't have to, you know,
[14:09.300 --> 14:14.860]  talk to somebody immediately. They'll just see the message later. We did eventually settle on
[14:14.860 --> 14:19.760]  having meetings every week, which is really, really useful to, you know, have these little
[14:19.760 --> 14:25.100]  iterations where you commit to something and get it done. But whenever we have an actual contest
[14:25.100 --> 14:31.320]  to run, we like to be all in the same place. So this is the rental house we had for our 2017
[14:31.320 --> 14:35.600]  qualifiers. And this is right before the game starts where we're figuring out what order
[14:35.600 --> 14:41.580]  challenges get unlocked. Once we get to Vegas, we find it really nice to have a shared hotel room
[14:41.580 --> 14:47.080]  where we set up all the noisy computers and stash all the snacks and stash all the drinks,
[14:47.080 --> 14:50.260]  including lots of water because it's important to stay hydrated.
[14:51.340 --> 14:55.020]  And, you know, that's kind of our home base when we're in Vegas.
[14:56.260 --> 15:01.240]  And the snacks and the drinks are really, really important because your team is a team made out
[15:01.240 --> 15:12.040]  of people. And people, you know, people require food and snacks. So in 2014, we were having computer
[15:12.040 --> 15:18.840]  problems. One of the team's instances, their actual physical hardware was acting really flaky
[15:18.840 --> 15:25.680]  and it needed to get unplugged and plugged back in a lot, like more than any other team should.
[15:25.740 --> 15:30.460]  And it was causing a delay of the game because processes would be trying to talk to that machine
[15:30.460 --> 15:36.840]  and it was just not on. So I was on stage trying to figure out why the scoreboard was going slow.
[15:36.840 --> 15:43.400]  And Salir was walking back and forth between his computer on stage to read logs and the rack of
[15:43.400 --> 15:50.280]  hardware off the stage to unplug and plug it back in to power cycle the machine. And I was kind of
[15:50.280 --> 15:55.740]  like getting impatient with how things were going. And I just kind of asked him, how much longer is
[15:55.740 --> 16:02.000]  this going to take? And whenever he walked off stage, he didn't go to the hardware. He just kind
[16:02.000 --> 16:11.220]  of left the room. And Salir is normally like the most stable, level-headed, friendliest person.
[16:11.220 --> 16:16.720]  And that's not something that ever happens. And that really, really scared the rest of us running
[16:16.720 --> 16:23.280]  this game. And eventually after about half an hour, he came back with a drink. He came back
[16:23.280 --> 16:29.000]  with a sandwich and he solved the problem basically instantly. And what I'd kind of
[16:29.000 --> 16:33.840]  neglected was to think about what he needed, you know, both in emotional terms and also
[16:33.840 --> 16:40.260]  it was lunchtime and that's why we were both kind of crabby. So supporting the CTF players
[16:40.860 --> 16:44.520]  means supporting the organizing team and making sure everybody's well fed.
[16:46.500 --> 16:51.540]  The final point I need to talk about with smooth operation is that Capture the Flag software is
[16:51.540 --> 16:58.000]  still software. And just like any kind of software or any other computer system, the way to make this
[16:58.000 --> 17:02.960]  software work reliably is really screwed up. By the end, we could have another machine spun up in
[17:02.960 --> 17:08.360]  about five minutes and we usually had hot spares ready to go anyways, which was great.
[17:09.660 --> 17:14.340]  So next up, I'm going to talk about running a fair contest. And a fair contest is really
[17:14.340 --> 17:21.020]  challenging because Capture the Flag is a game about computer hacking, but DEF CON CTF is also
[17:21.160 --> 17:26.940]  a computer system, which means that we have to be really, really paranoid. And we have to worry
[17:26.940 --> 17:33.640]  about teams trying to hack the right service the wrong way. We also want to make sure that teams
[17:33.640 --> 17:38.080]  don't hack the wrong thing. We want them to hack a vulnerable service. We don't want them to attack
[17:38.080 --> 17:45.540]  the scoreboard. And for attack defense games, we have another problem where teams may want to fix
[17:46.360 --> 17:53.580]  a broken software the wrong way. And the way we eventually, you know, solve this is by restricting
[17:53.580 --> 17:59.780]  what players can do more. So for qualifiers, we tend to run services on separate hosts.
[17:59.780 --> 18:03.940]  And we eventually got to running multiple hosts in multiple locations.
[18:04.280 --> 18:10.600]  So for the different challenges we have in qualifiers, each one would get at least one
[18:10.600 --> 18:17.100]  machine per data center. So teams in the U.S. would connect to a data center in North Virginia
[18:17.100 --> 18:23.500]  or something. Teams in Europe would go to Ireland, Asia Pacific, wherever Amazon's Asia Pacific hub
[18:23.500 --> 18:28.960]  is, which was great. Teams got lower ping times and we didn't have to worry about the entire
[18:28.960 --> 18:36.980]  service getting screwed up globally. Next, each connection to a vulnerable service would get a
[18:36.980 --> 18:43.300]  separate container using the run C container run time and xinitd to actually manage the connections.
[18:43.300 --> 18:48.820]  Using xinitd also has a really nice benefit where instead of writing each service having to have
[18:48.820 --> 18:53.380]  its own, like, network handling code, we could write all these services as just standard in and
[18:53.380 --> 19:00.940]  out executables. And then finally, in 2015, what we started doing was limiting what system calls
[19:00.940 --> 19:07.740]  each service would be able to make using the set comp tools in Linux. Finals is a much more
[19:07.740 --> 19:13.260]  complex game with much more complex problems. And one of the problems that we had was we kept
[19:13.260 --> 19:18.640]  trying to keep the game about reverse engineering and not a game about operating system administration.
[19:19.560 --> 19:24.760]  Starting in 2013, what we did was each team, whenever they connected to their machine,
[19:24.760 --> 19:29.860]  instead of getting root or super user access, they'd get an unprivileged team account that all
[19:29.860 --> 19:35.180]  it could really do is overwrite the service binaries that then ran under their own accounts.
[19:35.800 --> 19:41.720]  So we had a problem starting that year, and it got really bad in 2014, and it was a phenomenon
[19:41.720 --> 19:47.320]  we called the Superman defense. After the comic book character who just, like, one of his special
[19:47.320 --> 19:53.380]  powers is, oh, he just doesn't get hurt ever. So some of the Superman defenses would be like a
[19:53.380 --> 19:58.320]  firewall rule that says this traffic coming from the scoring system is okay, let it through,
[19:58.320 --> 20:04.040]  but this traffic coming from an opponent team is not okay, so just block it. And we never let teams
[20:04.040 --> 20:09.840]  have firewalls that way. However, a Superman defense that was very successful was preventing
[20:09.840 --> 20:15.780]  the flag being read from disk. So teams would run the service in an emulator, and the emulator
[20:15.780 --> 20:22.180]  wouldn't have flag access, and that was that. And we tried to solve that with making legitimate
[20:22.180 --> 20:28.780]  pollers read the flag, but it didn't always fit into a service nicely. So what we learned
[20:29.400 --> 20:35.960]  was that the U.S. Defense Advanced Research Projects Agency, or DARPA, started this project
[20:35.960 --> 20:42.440]  called Cyber Grand Challenge in 2014. And one of their goals was to, or their one goal, was to
[20:42.440 --> 20:49.380]  build a capture the flag game for autonomous computers with extremely formal rules. So for
[20:49.380 --> 20:54.000]  their thing, instead of having, you know, just services on a disk that would run, they came up
[20:54.000 --> 21:01.860]  with new jargon for it called challenge binaries, or CBs. These CBs were all 32-bit x86 binaries,
[21:01.860 --> 21:07.660]  but they used a special executable format that had limited system calls and couldn't store state.
[21:07.820 --> 21:11.500]  So what this meant was that teams couldn't just break out of the jail,
[21:11.500 --> 21:17.240]  or they couldn't wrap it in an emulator. They had to solve these things the, quote, right way.
[21:18.900 --> 21:25.320]  Similar to that, instead of just launching exploits over a network, CGC gave us this idea of a proof
[21:25.320 --> 21:30.780]  of vulnerability, which was the same kind of executable, but it had a few extra functions
[21:30.780 --> 21:35.240]  it could do. It could declare that it was going to demonstrate that it could control registers
[21:35.240 --> 21:40.720]  of a target program, or leak memory from the target program, and the scoring system instead
[21:40.720 --> 21:47.060]  of the teams managed how they ran. So these were put in place to support this goal of
[21:47.060 --> 21:53.000]  offline evaluation, where teams connect to a website and they get the binaries, the challenge
[21:53.000 --> 21:58.400]  binaries that other teams are using. And whenever they go to patch it, they upload it to the team
[21:58.400 --> 22:02.980]  interface website again. Whenever they have an attack, they write a proof of vulnerability
[22:02.980 --> 22:09.100]  and submit it. So the scoring system then runs the availability checks and proofs of vulnerability
[22:09.100 --> 22:15.040]  and isolation. And the value there is that we can run those offline, or we can run them later
[22:15.040 --> 22:19.160]  in a way that's designed to be reproducible and auditable.
[22:20.940 --> 22:26.380]  So once we learned about that, in finals, we could bring some of that technology to our game.
[22:26.420 --> 22:33.860]  Starting in 2015, we would restrict what system calls vulnerable services could make. And in 2016,
[22:33.860 --> 22:39.180]  the day after the Cyber Grand Challenge, we ran our game using their same format, with challenge
[22:39.180 --> 22:46.300]  binaries with proofs of vulnerability. And then finally, for our last year, we ran the entire game
[22:46.300 --> 22:51.560]  in a limited emulator that didn't have a lot of hooks into the messy real world of Linux.
[22:54.280 --> 22:58.980]  Another part of our fair contest is releasing the scoring information after the game's over.
[22:59.320 --> 23:05.580]  For qualifiers, we release the, you know, scoring information and a little bit of that.
[23:05.580 --> 23:12.700]  For finals, we release just, here's the database dump from Postgres, here's all the binaries teams
[23:12.700 --> 23:17.680]  uploaded, here's packet captures for the entire game. And that meant that a lot of teams could
[23:17.680 --> 23:23.320]  spend time after the game analyzing what was being used against them. Third parties could
[23:23.320 --> 23:27.300]  audit the scores and figure out how the actual scoring system in the game worked.
[23:30.360 --> 23:35.240]  An important part of running a fair contest is thinking about accessibility. And accessibility
[23:35.240 --> 23:38.640]  can mean a lot of things, and all these things are important.
[23:39.040 --> 23:43.760]  Our first year, we had a challenge called Die Hard, which required sensitive timing.
[23:43.880 --> 23:48.500]  This was a problem because the name of the challenge was supposed to be a clue,
[23:48.500 --> 23:54.580]  based on a popular movie in the US, about what players had to do. And the timing was difficult
[23:54.580 --> 23:59.760]  if you were on a, you know, internet connection that wasn't super close to where the game was
[23:59.760 --> 24:06.120]  being run. And what we had to do in future years, well, that year we, you know, had a really bad
[24:06.120 --> 24:12.740]  solution. We told teams to rent space near our servers and play from there and write a program,
[24:12.740 --> 24:17.740]  which is like, I don't want to be in the middle of a CTF game renting new servers.
[24:18.800 --> 24:23.500]  Long term, what we had to do was we had to rethink how a lot of our challenges were being built.
[24:23.500 --> 24:31.020]  We had to remove this idea that, you know, teams should be familiar in, you know, United States
[24:31.020 --> 24:37.220]  online, you know, hacker culture from our game. We had to eliminate the whole idea of precise timing
[24:37.220 --> 24:43.780]  for our game. And I think our game improved for that. Some of the other accessibility issues are
[24:43.780 --> 24:50.140]  just the languages we use online in our official communication. Because not everybody's a, you know,
[24:50.140 --> 24:55.660]  the same language as you are. And a lot of people are going to use machine translation
[24:55.660 --> 25:01.540]  because it works. But you have to, you know, use shorter sentences, use smaller words.
[25:01.600 --> 25:07.380]  And it's difficult to get in that frame of mind. And I don't know if we did a great job, but we
[25:07.380 --> 25:14.540]  thought about it at least. So, the last aspect of Capture the Flag and what I've learned running it
[25:14.540 --> 25:20.500]  is this idea of fun challenges. And the fun challenges that we spent a lot of time on,
[25:20.500 --> 25:24.780]  that we think were the most successful, we went and broke expectations.
[25:26.140 --> 25:32.960]  One of the ones from 2015 was a challenge called DOS Fun For You. And there's one writeup for this
[25:32.960 --> 25:39.680]  online. And the writeup starts with discover this challenge is a DOS binary, the disk operating
[25:39.680 --> 25:47.960]  system from the 80s. And they discovered that the IDA Pro disassembly tool was doing it wrong.
[25:47.960 --> 25:52.700]  So, they ended up debugging their disassembler and patching it before they could actually start
[25:52.700 --> 25:58.140]  the reverse engineering. And some teams found out that they had to use the emulator we told them to
[25:58.140 --> 26:03.580]  use because if they used something else, if they used DOS box, it would get some of the subtle
[26:03.580 --> 26:12.160]  timing wrong and they'd have a bad time. Another fun one was our Badger challenge from 2015 finals.
[26:12.160 --> 26:18.840]  And what it was, was a service running on MSP430 microcontroller on physical hardware
[26:19.360 --> 26:25.480]  that communicated with other teams with the game network using a custom CDMA radio network.
[26:25.660 --> 26:31.960]  So, this is it right here. We had two versions, I think three different versions of this badge.
[26:31.960 --> 26:38.160]  We had the one for competing teams. We had the one for organizers and VIPs that showed game scores.
[26:38.160 --> 26:42.960]  We had the base station which was hung up on a string over the game area and had a bunch of
[26:42.960 --> 26:50.620]  wires hanging out of it. So, one thing that we learned from Cyber Grand Challenge was this idea
[26:50.620 --> 26:57.580]  of consensus evaluation. So, with most with a traditional attack defense game, whenever you
[26:57.580 --> 27:03.280]  patch a binary on your machine, your opponents don't get to see how the binary has been patched,
[27:03.280 --> 27:08.260]  they just notice that their attacks don't work anymore. And what CGC did was they introduced
[27:08.260 --> 27:13.480]  this idea called consensus evaluation, where whenever you patch or you replace a binary with
[27:13.620 --> 27:20.680]  a fixed one, everybody else gets to see that. Which means that if your patch isn't complete,
[27:20.680 --> 27:26.620]  other teams, you know, can still attack it. And it also means that if your patch is really good,
[27:26.620 --> 27:32.860]  other teams will start using it. But the important part of that is that now, instead of us releasing
[27:32.860 --> 27:37.760]  eight services over an entire weekend and only getting eight binaries, every time somebody
[27:37.760 --> 27:44.260]  patches it, there's a new binary that needs reversing. So, what I wanted to do with these
[27:44.260 --> 27:49.400]  quals challenges was I wanted to push teams into this idea that automated program analysis
[27:50.600 --> 27:56.500]  is important. You can't just, you know, open up IDA Pro, solve a binary, and then, you know,
[27:56.500 --> 28:01.360]  you're done. You need to start thinking about programs, not just as software that runs,
[28:01.360 --> 28:08.980]  but as input for other programs. So, I worked on this thousand cuts category in 2015,
[28:08.980 --> 28:16.320]  16, and on a category called Crack Me 2000 in 2017. Each of these challenges had hundreds
[28:16.320 --> 28:22.560]  of binaries that required automated analysis. And I feel they were successful. I saw a lot
[28:22.560 --> 28:26.980]  of blog posts and a lot of write-ups about how to solve these that mentioned they'd never done
[28:26.980 --> 28:34.800]  automated program analysis before. Consensus evaluation and finals gave us some really funny
[28:34.800 --> 28:40.420]  moments. So, a player from a team that I'm going to refrain from naming walked up to the organizer's
[28:40.420 --> 28:45.520]  table and asked why they were losing points. You know, their service was being attacked.
[28:46.040 --> 28:49.380]  We looked at the scoreboard. It's like, yeah, your service is being attacked. That's why you're
[28:49.380 --> 28:54.560]  losing points. It's called capture the flag. But they claimed they were using the same binaries
[28:54.560 --> 29:00.600]  as the winning team and that it couldn't be that. And we just kind of, like, looked at them.
[29:02.720 --> 29:09.060]  And then they just kind of did this and walked away. Because what they realized was that the
[29:09.060 --> 29:15.720]  winning team had put a secret code in their, or a backdoor in their binary to allow them to attack
[29:15.720 --> 29:22.320]  anybody that didn't, that just used their binary without thinking about it. That was really funny.
[29:22.320 --> 29:26.920]  You know, we still laugh about just the expression on that guy's face all the time.
[29:28.400 --> 29:34.740]  So, the Rubik's service from 2017. A few minutes ago I mentioned how it worked. You upload the
[29:34.740 --> 29:40.080]  instructions to rotate a Rubik's cube and then if you've done it successfully, those instructions
[29:40.080 --> 29:49.120]  get evaluated as just CPU byte code. So, how this worked in practice was teams would start to
[29:49.120 --> 29:57.360]  write code to block the evil, whenever shellcode would come in, the scoring system would submit
[29:57.360 --> 30:02.700]  some that didn't steal the flag, but attacking players would submit shellcode that did try and
[30:02.700 --> 30:08.260]  steal the flag. So, defenders would start using the extra space in this binary and they'd patch
[30:08.260 --> 30:13.600]  it by saying this shellcode let it through, this shellcode block it, this code let it through,
[30:13.600 --> 30:19.280]  this code block it. So, what that meant was whenever a team would patch Rubik's, somebody
[30:19.280 --> 30:24.820]  that was already successfully attacking them would disassemble the binary, see their previous attack
[30:24.820 --> 30:30.780]  reflected in it, and have to build a new attack, which felt like the comments we got after the
[30:30.780 --> 30:35.540]  game were that this felt like a multiplayer game against other people. We were having,
[30:35.540 --> 30:39.980]  I was having a back and forth with somebody on this other team. They'd patch it, I'd write a
[30:39.980 --> 30:45.580]  new exploit. They'd patch it, I'd write a new exploit. And we also had to update the scoring
[30:45.580 --> 30:52.020]  system to, you know, create new good shellcode. But it was a lot of fun for the players and it
[30:53.600 --> 31:00.240]  just fascinates me. So, we've talked about running smoothly, we've talked about running a fair
[31:00.240 --> 31:05.900]  contest, and we've talked about fun challenges. This is not everything that we know about
[31:05.900 --> 31:10.040]  Capture the Flag, and this is not everything that there is to know about Capture the Flag.
[31:10.320 --> 31:14.880]  There's still a lot more to learn, and there's a lot more work ahead of us building new Capture
[31:14.880 --> 31:23.320]  the Flag games that create new opportunities for new players. CTF is not a huge community yet.
[31:24.000 --> 31:27.360]  How many people in here have never played a Capture the Flag game?
[31:29.200 --> 31:35.260]  Okay, that's actually really impressive. But I think in a lot of the world, there's still a lot
[31:35.260 --> 31:43.500]  more work that we can do to make Capture the Flag for new players. The best way to learn how to do
[31:43.500 --> 31:49.520]  Capture the Flag is to do it. Either play a Capture the Flag game, or once you're happy with that,
[31:49.520 --> 31:54.380]  build your own Capture the Flag game for your friends and for people that you like to play.
[31:56.000 --> 32:00.920]  Capture the Flag has given me five years with the best group of people I've ever worked with,
[32:00.920 --> 32:07.460]  and it's given me an amazing five years building a contest for the CTF community around the world,
[32:07.460 --> 32:13.420]  the friendliest and smartest community I know. Thank you for making it amazing. And if you're
[32:13.420 --> 32:19.220]  curious about DEF CON Capture the Flag, order The Overflow, the new DEF CON CTF organizers.
[32:19.220 --> 32:23.000]  They're starting their game tomorrow morning at 8am Beijing time.
[32:23.000 --> 32:27.160]  You can learn more about it at oooverflow.io
